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Abstract 

For  proper  operation  of  today’s  large  digital  private  networks,  high  quality  synchronization  must 
be  achieved.  A  general  telecommunication  performance  objective  is  to  maintain  long-term  frequency 
accuracy  of  ten  parts  per  trillion  at  all  synchronous  digital  equipment  in  the  network.  Many  times, 
however ;  this  is  not  achieved  in  private  networks.  Low  quality  clocks,  errored  transmission  facilities, 
and  incorrectly  designed  synchronization  plans  are  often  cause  for  poor  performance.  It  is  shown 
that  properly -designed  private  networks  can  operate  with  long  term  frequency  averages  between  ten 
parts  per  trillion  to  ten  parts  per  million. 

These  performance  levels  can  adversely  impact  customer  applications.  The  most  demanding 
applications  are  digital  and  voiceband  data,  encrypted  voice,  facsimile  and  video.  In  a  typical 
private  network  operating  at  0.01  parts  per  million,  the  user  would  experience  reduced  data 
throughput,  dropped  encrypted  calls,  unreadable  facsimile  pages,  or  interrupted  video  transmission 
dozens  of  times  per  day. 

The  major  contribution  to  poor  private  network  synchronization  performance  is  the  interaction 
of  Customer  Premises  Equipment  (CPE)  clocks  and  the  network  facilities  used  to  distribute 
timing.  The  performance  of  typical  CPE  clocks  and  facilities,  and  their  impact  on  customer 
applications,  are  discussed.  CPE  clock  performance  issues,  along  with  private  network  architectural 
constraints,  make  synchronization  planning  extremely  difficult.  Planning  is  usually  costly  and 
requires  specialized  expertise. 


CUSTOMER  EXPECTATIONS 

Intelligent  digital  networks  which,  can  consist  of  the  public  switched  network’s  facilities  and  services, 
as  well  as  customer  owned  equipment  must  provide  nearly  error  free  performance  under  a  variety 
of  operating  conditions.  This  includes  automatic  reconfiguration  of  the  network  (which  may  occur 
under  software  control,  without  operator  intervention)  during  normal  operations  and  failure  modes. 

The  robustness  of  a  network  can  be  improved  against  observable  degradation  of  services  during 
failure  modes,  if  a  quality  synchronization  plan  is  in  place.  To  attain  such  a  plan,  one  must  consider 
synchronization  performance  at  each  level  in  the  network  including: 


1 .  The  synchronization  performance  of  the  public  switched  network, 

2.  The  private  network’s  synchronization  architecture,  and 
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3.  The  performance  of  individual,  CPE  clocking  systems. 


AT&T  has  strived  to  improve  the  synchronization  performance  of  both  the  public  switched  network 
and  CPE  [1-5].  Thus,  customers  can  expect  and  get  the  best  possible  error-free  performance  from 
digital  networks. 

NEED  FOR  SYNCHRONIZATION  IN  DIGITAL  NETWORKS 

Synchronization  is  the  means  used  to  keep  all  the  equipment  in  a  digital  telecommunications  net¬ 
work  operating  at  the  same  average  rate.  If  the  receiving  end  is  to  receive  and  properly  interpret 
the  digital  signals  transmitted  over  a  communications  link,  it  must  stay  in  step,  or  be  synchronized, 
with  the  transmitting  end.  When  digital  signals  are  transmitted  over  a  network  of  digital  com¬ 
munications  links,  switching  nodes,  and  transmission  interfaces,  all  entities  must  be  synchronized. 
This  is  known  as  network  synchronization  [5]. 

We  refer  to  the  loss  or  repetition  of  an  entire  frame  of  data  as  a  slip.  (The  loss  or  repetition  of 
an  entire  frame  of  data  along  frame  boundaries  is  more  accurately  called  a  controlled  slip.  For 
purposes  of  this  paper,  slips  and  controlled  slips  are  synonymous).  Slips  are  caused  by  differences 
in  clock  rates  within  a  network  and  by  the  effect  of  transmission- induced  impairments  on  clocks. 

The  primary  methods  used  to  control  the  slip  rate  and  limit  it  to  an  acceptable  level  are: 

1.  Operate  all  clocks  in  the  network  at  the  same  average  rate. 

2.  Place  buffers  at  the  receiving  end  to  absorb  small  phase  variations. 

In  the  United  States,  equipment  clocks  are  categorized  by  ANSI  standards  according  to  accuracy 
and  performance  level  [4].  Stratum  1  clocks  have  the  highest  accuracy  requirements:  stratum 
2,  3,  and  4  clocks  have  progressively  lower  accuracy  and  performance  requirements  [5].  Timing 
references  are  passed  hierarchically  from  high-performing  clocking  systems  to  equivalent  or  lower 
performing  clocking  systems  within  a  network  to  synchronize  many  equipment  clocks  to  one,  nom¬ 
inal  reference  frequency.  Most  of  the  clocking  systems  used  in  the  AT&T  switched  network  are 
stratum  1  and  stratum  2.  Most  clocks  found  in  CPE,  such  as  PBX’s  and  multiplexers,  are  stratum 
4. 

However,  even  when  all  the  equipment  clocks  are  synchronized  to  a  single  master  clock,  it  still  is 
impossible  to  eliminate  slips  in  a  digital  network.  Therefore,  synchronization  plans  are  developed 
to  minimize  and  control  the  rate  at  which  slips  occur.  A  synchronization  plan  specifies  the  timing 
flow  through  the  network  and  takes  into  account  equipment  performance,  facility  performance, 
and  network  topology. 

PRIVATE  NETWORK  SYNCHRONIZATION  PLANNING 

Customer  digital  private  networks  present  additional  complexities  for  synchronization  planning. 
Typically,  these  networks  have  diverse  architectures;  i.e.,  the  equipment  and  facilities  they  contain 
come  from  multiple  vendors  and  carriers  and  are  interconnected  in  complex  arrangements  [7] . 
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In  addition,  the  quality  of  the  clocks  in  CPE  tends  to  be  poor  in  private  networks.  Most  CPE 
currently  use  stratum  4  clocks,  which  are  not  designed  to  transfer  timing  references  between 
equipment.  Consequently,  the  private  network  arena  is  the  most  difficult  one  in  which  to  obtain 
high  quality  synchronization  performance  and  is  the  one  most  susceptible  to  overall  performance 
problems. 

SYNCHRONIZATION  ARCHITECTURE 

Ideally,  timing  flow  in  digital  private  networks  follows  the  hierarchical  source  to  receiver  method. 
That  is,  synchronization  references  are  passed  from  high  performing  clocks  to  the  low  performing 
clocks.  While  most  private  network  equipment  is  usually  equipped  with  stratum  4  clocking  systems, 
some  CPE  have  stratum  3  systems. 

Most  public  switched  network  services  contain  traceable,  stratum  1  timing  (i.e.  timing  that  origi¬ 
nates  from  a  stratum  1  primary  source) .  Hence,  if  the  timing  source  in  a  private  network  originates 
from  a  traceable,  stratum  1,  public  switched  network  link,  therl  such  timing  should  be  distributed 
to  all  CPE  throughout  the  network.  Ideally,  intra-building  timing  will  be  distributed  via  a  hi¬ 
erarchical  arrangement.  In  such  an  architecture,  the  best  performing  clock  in  a  building  is  used 
to  distribute  timing  to  all  office  equipment  via  dedicated  lines  in  the  office.  The  clock  is  referred 
to  as  a  BITS  (Building  Integrated  Timing  Supply).  The  BITS  receives  its  timing  reference  from 
outside  the  office.  This  architecture  is  used  for  the  AT&T  public  network.  In  private  networks,  a 
BITS  type  arrangement  is  unlikely  to  happen  because  most  CPE  do  not  have  a  dedicated  timing 
interface  and  can  only  derive  timing  from  a  traffic-carrying  signal. 

Figure  1  illustrates  a  typical  private  network  location.  This  location  is  connected  to  four  other 
locations  in  the  private  network,  as  well  as  to  three  public-switched-network  services.  Each  public- 
switched  network  service  contains  timing  that  can  be  traced  to  a  stratum  1  clock.  All  equipment 
at  the  location  requires  synchronization,  and  the  diagram  shows  a  reasonable  synchronization  plan 
for  the  site.  This  plan  adheres  to  the  hierarchical  timing-flow  requirements  and  permits  diverse 
timing  sources  into  the  location. 

However,  the  location  does  not  contain  a  true  BITS  architecture.  Instead,  the  location  illustrated 
in  Figure  1  has  two  BITS  clocks:  the  PBX,  and  the  digital  cross-connect  system  (DCS).  A  BITS 
architecture  offers  several  advantages: 

1.  Maximum  use  of  the  highest  performing  clock  at  the  location  (without  intralocation  cascading 
of  timing  references) 

2.  The  BITS  clock  serves  as  a  central  point  for  maintaining  synchronization  within  the  location. 


Because  of  implementation  problems,  private  network  BITS  designs  are  unlikely.  Customer  premises 
lack  bridging  repeaters  for  routing  timing  signals  within  the  site.  In  addition,  some  stratum  3  and 
most  stratum  4  CPE  systems  lack  dedicated  timing  interfaces. 

A  major  focus  for  any  synchronization  plan  is  to  overcome  topological  complexity  and  constraints. 
Private  networks  can  suffer  from  excessive  cascading  of  CPE,  lack  of  diversity  of  timing  references, 
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and  hierarchical  conflicts.  There  may  be  limited  connectivity  within  the  private  network  and  to 
public  switched  networks.  They  may  lack  provisioning  for  BITS  architectures. 

Before  a  good  synchronization  plan  can  be  developed  for  a  given  network,  we  must  consider  these 
issues  in  detail.  But  we  must  understand  that  these  problems  may  be  compounded  because  equip¬ 
ment  and  facility  performance  will  also  vary  and  affect  network  performance.  The  stratum  4  clock 
is  one  of  the  consistently  poor  performers  in  private  networks  [7], 

CPE  CLOCKS 

Originally,  stratum  4  clocks  acted  as  the  CPE  receive  clock,  which  terminates  timing;  they  were 
not  designed  to  transfer  timing  references  between  systems.  But  as  the  private  customer-owned 
networks  have  grown  more  complex,  the  clocks  in  CPE  (such  as  PBX’s  and  multiplexers)  have  been 
used  to  transfer  timing.  Private  networks  often  have  long  chains  of  stratum  4  systems  through 
which  timing  is  cascaded. 

The  main  problem  with  stratum  4  clocks  is  not  their  low  oscillator  accuracy  (typically  32  parts 
per  million),  but  rather  their  inability  to  switch  cleanly  between  timing  references.  If  the  timing 
reference  for  a  stratum  4  clock, is  sufficiently  impaired,  the  clock  will  switch  to  its  backup  timing 
source,  which  may  be  either  another  timing  reference  or  the  clock’s  internal  oscillator.  Stratum  4 
clocks  do  not  have  a  phase  build-out  mechanism  that  would  slowly  correct  for  the  phase  difference 
between  the  previous  and  current  sources  of  timing.  Therefore,  the  reference  switch  would  cause 
a  phase  hit  at  the  CPE’s  digital  interfaces.  If  this  CPE  is  providing  a  timing  reference  to  another 
piece  of  equipment,  the  receiving  CPE  will  react  to  this  degradation  (which  looks  like  an  error 
burst)  by  inducing  a  reference  switch  of  its  own.  Thus,  these  impairments  will  propagate  through 
the  chain  of  cascaded  stratum  4  clocks,  seriously  affecting  the  network’s  performance  [8]. 

Stratum  3  and  stratum  2  clocks  do  not  suffer  from  this  phenomenon.  If  a  reference  switch  occurs, 
these  clock  systems  correct  for  phase  differences  between  references  [6]. 

Today,  digital  network  standards  define  a  new  level  of  stratum  4  clock  that  incorporates  phase 
build-out  when  references  are  switched.  For  example,  AT&T’s  TR  62411  (the  interface  specifica¬ 
tion  for  Accunet  T1.5  service),  which  was  last  published  in  December  1988,  defined  an  enhanced 
stratum  4  clock  that  can  be  used  for  timing  transfer  [8].  This  clock,  which  TR  62411  calls  stra¬ 
tum  4,  type  I,  met  the  requirements  for  maximum  time-interval  error  (MTIE),  which  regulates 
the  total  phase  movement  and  the  rate  of  change  of  phase  movement  of  the  timing  signal  under 
reference-switching  conditions.  In  addition,  a  new  EIA/TIA  standard  defines  the  same  enhanced 
stratum  4  system  [10]. 

PRIVATE  NETWORK  PERFORMANCE 

Overall,  the  typical  synchronization  performance  of  a  digital  private  network  can  be  up  to  100,000 
times  worse  than  the  performance  of  the  AT&T  network  (in  terms  of  frequency  accuracy).  When 
new  CPE  synchronization  standards  for  the  telecommunications  industry  are  implemented  in  pri¬ 
vate  networks,  this  performance  should  improve  by  several  orders  of  magnitude.  Generally,  a 
high-quality  synchronization  plan  is  needed  to  minimize  the  occurrence  of  slips,  error  bursts,  and 
phase  hits  in  private  networks. 
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SYNCHRONIZATION  PLAN  DESIGN  CONCEPTS 

Some  basic  concepts  need  to  be  considered  during  the  design  of  a  synchronization  plan: 


1.  Whenever  possible,  it  is  important  to  adhere  to  the  hierarchical  rules  of  timing  transfer. 
That  is,  timing  should  always  be  passed  from  low  numbered  stratum  devices  (e.g.  stratum 
1,  stratum  2)  to  equivalent  or  higher  numbered  stratum  devices  (e.g.  stratum  3,  stratum  4). 

2.  Traceable,  stratum  1  timing  sources  need  to  be  used  whenever  possible,  and  timing  sources 
should  be  diverse  whenever  possible. 

3.  Timing  loops  must  be  avoided.  These  loops  occur  when  two  or  more  CPE  transfer  timing  to 
each  other  and  form  a  loop  without  a  designated  master  source. 

4.  The  cascading  of  timing  references  through  CPE  should  be  minimized. 


These  are  some  of  the  fundamental  concepts  that  are  the  basis  of  over  6000  rules  that  AT&T  has 
developed  for  synchronization  planning  of  digital  private  networks.  The  rules  have  been  incorpo¬ 
rated  into  an  expert  system,  called  PRINCE-S,  that  was  developed  by  AT&T  Bell  Laboratories. 
PRINCE-S,  stands  for  PRIvate  Network  Computer-based  Expert-system  for  Synchronization  [9], 

EXPERT  SYSTEM  FOR  SYNCHRONIZATION  PLANNING 

AT&T  developed  the  PRINCE-S  system  to  mechanize  the  task  of  synchronization  planning  for 
a  private  network  in  a  consistent,  accurate,  and  reliable  way.  AT&T  synchronization  engineers 
have  used  the  system  to  help  them  design  synchronization  plans  and  maintain  synchronization 
networks. 

The  system  generates  network  maps  that  specify  primary  and  backup  timing  reference  distribution 
for  each  CPE,  and  estimates  the  synchronization  performance  of  the  network  end-to-end  (and 
for  each  CPE).  The  system’s  recommendation  represents  the  best  synchronization  performance 
attainable,  based  on  information  used  to  model  the  network  topology,  equipment,  facilities,  and 
public-switched-network  connectivity. 

SUMMARY 

We  have  explained  how  AT&T  provides  the  means  to  give  the  best  possible  synchronization  perfor¬ 
mance  in  digital  customer-owned  private  networks.  This  includes  consideration  of  synchronization 
planning  for  customer-owned  networks  and  CPE  clock-performance  issues. 
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. DSX  connection 

■  Primary  reference 

- Backup  reference 


Figure  1.  Synchronization  plan  for  all  equipment  at  a  representative  location  in  a  private 
network.  This  location  is  connected  to  four  other  locations  in  the  network  and  to  three 
public-switched-network  services.  The  timing  for  each  network  service  can  be  traced  to  a 
stratum  1  clock. 
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QUESTIONS  AND  ANSWERS 

D.  Allan,  Allan’s  Time:  This  is  a  broader  question  than  just  a  private  network  but  perhaps  you 
can  address  it.  If  the  timing  community  could  provide  for  telecommunication  a  clock  or  clock 
system  which  would  allow  you  to  synchronize  the  network  with  even  a  broad  network  say  at  the 
100  nanosecond  level  forever  for  all  the  nodes.  What  would  that  mean  to  telecommunication, 
if  it  could  be  done  inexpensively? 

C.  Near,  AT&T  Bell  Laboratories:  It  depends  upon  how  inexpensive  is  inexpensive.  It  is  in 
a  normal  carrier  environment,  for  long  distance  carriers  or  your  locals  where  they  can  afford 
clocking  systems,  that  could  be  of  benefit.  If  they  were  in  a  few  (?)  range,  that  is  fine. 
Private  networks  is  where  you  are  going  to  be  getting  a  much  larger  customer  basis.  They  are 
not  going  to  spend  that  much  money.  They  will  spend  maybe  a  $1000.00  for  a  clock  and  will 
not  want  to  spend  a  whole  lot  more  than  that.  Sometimes  you  can  get  them  to  spend  maybe 
$5000.00  on  a  clock  but  those  are  cases  where  you  are  lucky.  If  it  is  in  a  few  hundred  to  a 
$1000.00  range  and  offer  that;  then  they  will  go  for  it. 

R.  Brown:  Just  my  two  cents  on  that  Dave.  The  one  thing  that  I  would  like  to  emphasize  a 
little  bit  from  Chris’s  talk  is  this  is  a  field  with  a  huge  number  of  players  in  it.  If  you  look  at 
our  specifications  I  think  you  people  might  say  that  performance  is  lousy,  these  guys  have  got 
to  be  dumb  if  they  can  not  achieve  this.  We  are  not  dumb,  it  is  just  we  have  a  lot  of  players. 
We  have  got  different  carriers,  we  have  got  different  vendors.  Any  circuit  that  is  going  to  go 
across  the  country,  you  have  got  local  exchange  carriers  and  inter-exchange  carriers.  Those 
carriers  buy  equipment  from  different  vendors  and  so  you  are  looking  at  a  chain  of  clocks  that 
is  managed  by  possibly  three  management  entities  and  the  equipment  might  be  built  by  a  dozen 
different  vendors.  It  might  be  possible  for  us  to  say  this  is  good  clock  design  but  we  can  not 
force  people  to  implement  a  design.  As  far  as  something  at  every  node,  our  clients  themselves 
are  looking  at  LORAN  receivers.  This  is  a  fairly  inexpensive  thing  that  might  be  implemented 
at  every  node.  Right  now  their  impression  is,  that  is  not  cost  effective.  Something  has  got  to 
be  cheaper  than  to  be  at  every  node. 

G.  Winkler,  USNO:  This  is  very  right  but  on  the  other  hand  if  you  have  a  huge  number 
of  customers  that  is  going  to  bring  the  price  down  tremendously.  Don  not  forget  all  the 
clock  technology  today  is  based  on  a  customer  base  which  can  absorb  maybe  several  thousand 
precision  clocks  per  year.  If  they  can  absorb  five  hundred  thousand  clocks  per  year,  the  price 
is  going  to  drop  as  precipitously  as  the  GPS  receivers  have  dropped.  Today,  only  three  years 
after  a  GPS  receiver  cost  $50,000.00,  it  is  down  to  $900.00.  I  would  predict  to  you  that  a  clock 
which  will  have  one  part  and  1011  capability  will  cost  you  less  than  $500.00  seven  years  from 
now,  if  you  have  that  kind  of  a  customer  base  and  if  you  will  completely  turn  around  all  these 
considerations. 

W.  Harding,  Naval  Electronics  Systems:  That  customer  base  is  just  right  around  a  corner  for 
you  because  with  the  advent  of  the  telephone  systems  and  the  pictures  that  you  are  going  to 
be  able  to  see  years  from  now  by  the  person  that  you  are  talking  to  on  the  other  end.  Also 
the  individual  customers  as  far  as  VHS  communications  will  all  be  through  satellites.  So  you 
are  going  to  have  a  big  base. 

R.  Kern,  Kemco:  I  have  a  question.  I  am  new  to  telecommunications  but  I  have  not  heard 
any  of  you  fellows  talk  about  reliability.  What  is  your  experience?  What  MTBF  do  you  need? 
You  can  talk  about  a  $100.00  but  if  it  is  only  going  to  last  a  week  you  are  in  trouble.  Could 
you  comment  on  reliability  or  have  you  guys  got  standards  on  that? 

R.  Brown:  I  am  going  to  comment  on  that  real  quickly  and  then  we  will  get  to  Bob’s  talk  and 
save  any  other  questions  to  the  end.  Just  on  reliability,  what  we  have  done  at  Bellcore  is  we 
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write  requirements  for  equipment.  We  require  a  Jot  of  duplication  of  equipment  so  that  no 
single  failure  will  cause  the  system  to  fail.  That  is  really  the  way  we  have  gone  after  reliability 
and  it  seems  to  have  worked  out  ok,  that  you  don  not  get  any  duplicated  failures.  As  far  as 
specific  MTBF  numbers,  the  clocks  and  the  function  they  are  serving  are  very  critical  so  it  has 
to  be  very  high.  The  way  we  have  attacked  it,  is  to  require  duplication  of  equipment. 
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